Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch shouldn't try to balance shards from nodes with newer version of lucene to nodes with older versions of lucene #4588

Closed
nik9000 opened this issue Jan 2, 2014 · 1 comment · Fixed by #4602

Comments

@nik9000
Copy link
Member

nik9000 commented Jan 2, 2014

During a rolling upgrade I'm constantly seeing IndexFormatTooNewException exceptions as Elasticsearch repeatedly tries to rebalance a shard from an upgraded machine to a non-upgraded machine. It is causing a large load spike on the non upgraded machine and won't work anyway. Can Elasticsearch just not try?

@ghost ghost assigned s1monw Jan 2, 2014
@s1monw
Copy link
Contributor

s1monw commented Jan 2, 2014

While I agree that we shouldn't do this and it's on my list of thing that I should have been doing months ago, I don't think it will fix the spikes. You should really disable allocations / relocations while you restart though. But lets fix this problem since it could potentially lead to other issues.

@s1monw s1monw closed this as completed in 5b5b2e6 Jan 3, 2014
s1monw added a commit that referenced this issue Jan 3, 2014
…e forward compatibility.

Today during restart scenarios it is possible that we recover from a node that
has already been upgraded to version N+1. The node that we relocate to is
on version N and might not be able to read the index format from the node
we relocate from. This causes `IndexFormatToNewException` during
recovery but only after recovery has finished which can cause large
load spikes during the upgrade period.

Closes #4588
brusic pushed a commit to brusic/elasticsearch that referenced this issue Jan 19, 2014
…e forward compatibility.

Today during restart scenarios it is possible that we recover from a node that
has already been upgraded to version N+1. The node that we relocate to is
on version N and might not be able to read the index format from the node
we relocate from. This causes `IndexFormatToNewException` during
recovery but only after recovery has finished which can cause large
load spikes during the upgrade period.

Closes elastic#4588
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
…e forward compatibility.

Today during restart scenarios it is possible that we recover from a node that
has already been upgraded to version N+1. The node that we relocate to is
on version N and might not be able to read the index format from the node
we relocate from. This causes `IndexFormatToNewException` during
recovery but only after recovery has finished which can cause large
load spikes during the upgrade period.

Closes elastic#4588
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants